Arrow signal recognition device

ABSTRACT

Based on an image captured by an onboard camera, an arrow signal detector sets an arrow signal area on the basis of a signal light distance between a lit red signal light of a traffic light and a vehicle equipped with the arrow signal recognition device, counts the number of color tone effective pixels assumed as being lit within each arrow signal area, further searches for and counts color tone ineffective pixels in the color tone effective pixels on the basis of pixel information on the vicinity of each color tone effective pixel, and calculates an arrow effective pixel number from the difference between the number of color tone effective pixels and the number of color tone ineffective pixels.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority from Japanese Patent ApplicationNo. 2013-271587 filed on Dec. 27, 2013, the entire contents of which arehereby incorporated by reference.

BACKGROUND

1. Technical Field

The present invention relates to an arrow signal recognition device thatrecognizes an arrow signal light provided to a traffic light, on thebasis of an image captured by an onboard camera.

2. Related Art

Conventionally, a technique is known in which a traffic light and anarrow signal light provided to the traffic light are recognized based ona driving environment ahead of a vehicle that is image-captured by acamera (hereinafter referred to as “onboard camera”) mounted to thevehicle.

For example, Japanese Unexamined Patent Application Publication No.2012-168592 discloses a technique of extracting an arrow signal light bydetecting a red signal of a traffic light based on an image captured byan onboard camera and setting a search region for the arrow signal lightbased on the position of the red signal within the image to detect apixel having a color feature amount of a predetermined RGB ratio withinthe set search region.

As traffic lights, there are light bulb traffic lights and LED trafficlights. Obviously, there are a light bulb type and an

LED type in arrow signal lights as well. In an LED arrow signal light,an arrow is formed by LEDs being arranged at predetermined intervals.Therefore, the luminance is approximately constant throughout. However,in a light bulb traffic light, the luminance gradually decreases awayfrom a light bulb, since a cover lens is colored in the color of lightwith a blue filter or the like. Further, the brightness of the lightbulb itself is not constant.

As a result, in the case where extraction of an arrow signal light isperformed with a threshold being a color feature amount of apredetermined RGB ratio as disclosed in the document mentioned above,recognition itself of the arrow signal light may be difficult from afarwhere a vehicle equipped with this technique is relatively distant froma traffic light, for example. As a solution, it is conceivable to set alow color feature amount. However, there is a problem that the precisionof recognition of an arrow signal light decreases.

SUMMARY OF THE INVENTION

In view of the circumstances described above, an object of the presentinvention is to provide an arrow signal recognition device that candetect an arrow signal light with high precision even from relativelyafar.

An aspect of the present invention provides an arrow signal recognitiondevice including: an onboard camera that captures an image of a drivingenvironment ahead of a vehicle equipped with the arrow signalrecognition device; an image recognition processor that recognizes a litsignal light provided to a traffic light and calculates a signal lightdistance between the lit signal light and the vehicle, on the basis ofone frame of the image captured by the onboard camera; and an arrowsignal detector that detects an arrow signal light provided to thetraffic light. The arrow signal detector includes: an arrow signal areasetting module that sets an arrow signal area with reference to the litsignal light on the basis of the signal light distance; a color toneeffective pixel searching module that searches for a color toneeffective pixel assumed as being lit from a color tone within an arrowsignal area set by the arrow signal area setting module and counts thenumber of the color tone effective pixels; a color tone ineffectivepixel searching module that compares each color tone effective pixelretrieved by the color tone effective pixel searching module and a pixelin a vicinity of the color tone effective pixel and then searches thecolor tone effective pixel for a color tone ineffective pixel andmoreover counts the number of the color tone ineffective pixels; anarrow effective pixel number calculating module that calculates an arroweffective pixel number by subtracting the number of color toneineffective pixels retrieved by the color tone ineffective pixelsearching module from the number of color tone effective pixelsretrieved by the color tone effective pixel searching module; and anarrow pixel determining module that determines an arrow pixel to bepresent in a case where the arrow effective pixel number calculated bythe arrow effective pixel number calculating module exceeds an arroweffective pixel threshold set based on the signal light distance.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic configuration diagram of an arrow signalrecognition device;

FIG. 2 is a first flowchart showing an arrow signal recognitionprocessing routine;

FIG. 3 is a second flowchart showing the arrow signal recognitionprocessing routine;

FIG. 4 is an illustration showing an image of a view ahead of a vehicleequipped with the an arrow signal recognition device that is captured byan onboard camera;

FIG. 5 is an illustration showing the positional relationship of atraffic light and the vehicle;

FIG. 6 is an illustration showing a search region for an arrow signallight that is set with reference to a red signal light;

FIG. 7A is an illustration of a state where an arrow signal light isprovided to a horizontal traffic light, and FIG. 7B is an illustrationof a state where an arrow signal light is provided to a vertical trafficlight;

FIG. 8A is an illustration in which color tone effective pixels areextracted from an arrow signal search region, and FIG. 8B is anillustration in which arrow effective pixels are extracted from colortone effective pixels; and

FIG. 9A is an illustration in which ineffective pixels are extractedfrom extracted arrow effective pixels based on the G edge differencerelative to pixels in the vicinity, and FIG. 9B is an illustration inwhich an ineffective pixel is extracted from an extracted arroweffective pixel based on pixels in the vicinity.

DETAILED DESCRIPTION

One example of the present invention will be described below withreference to the drawings. Reference numeral 1 in FIG. 1 denotes avehicle such as an automobile. In a front portion in the interior of thevehicle 1, an onboard camera 2 is placed in an upper portion of a frontwindshield. The onboard camera 2 is a stereo camera formed of a maincamera 2 a and a sub camera 2 b. The respective cameras 2 a and 2 b arebuilt in with an imaging element (image sensor) such as a CCD or CMOSand are arranged, for example, in symmetrical positions on the left andright across a rear-view mirror arranged in the middle in the vehiclewidth direction. Hereinafter, the onboard camera 2 refers to both themain camera 2 a and the sub camera 2 b for the sake of convenience,unless otherwise stated.

The onboard camera 2 is a color camera that captures a color image of adriving environment of a path on which the vehicle 1 is travelling.Pixel data of each imaging element is set with a luminance value (pixelvalue) of 8 bits (tone from 0 to 255) for each of R, G, and B. Theluminance value is not limited to that of 8 bits.

The main camera 2 a captures a reference image (right image) necessaryupon performing stereo processing, and the sub camera 2 b captures acomparison image (left image) for the processing. In a state of beingsynchronized with each other, respective analog images of R, G, and Boutput from the two cameras 2 a and 2 b are converted to a digital imageof a predetermined tone by an A/D converter 3. The digitally convertedimage is output to an image recognition unit 11.

The image recognition unit 11 is mainly configured of a microcomputerand includes a known CPU, ROM, RAM, and the like. For an arrow signalrecognition function processed in the CPU, an image recognitionprocessor 12, a camera controller 13, and an arrow signal recognizer 14serving as the arrow signal detector of the present invention areincluded. Further, the image recognition unit 11 includes a storage 15.The storage 15 stores respective pieces of data such values of as anarrow effective pixel number YOK detected for every arrow signal areadescribed later, an arrow effective point PNiOK and an arrow ineffectivepoint PNiNG of which one is updated for every frame, and an arrow signallight determination flag FNi set for every arrow signal area.

The image recognition processor 12 recognizes a three-dimensional objectahead of the vehicle 1 and calculates an average luminance value, basedon image data captured by the onboard camera 2. That is, as shown inFIG. 4 for example, a preceding car 22, a traffic sign 23, a white line24, a curb 25, as well as an oncoming car, pedestrian, or the like thatis not shown are three-dimensional objects other than a traffic light 21in image data captured by the onboard camera 2. These are recognized asrespective three-dimensional objects, and the distances between theseand the vehicle 1 are obtained from parallax in the main camera 2 a andthe sub camera 2 b. Recognition of the respective three-dimensionalobjects is performed using a known pattern matching method or the like.How the distance between a three-dimensional object and the vehicle 1 isobtained is known, and therefore description is omitted.

In the image shown in FIG. 4, the traffic light 21 includes a red signallight 21 r, a yellow signal light 21 y, and a green signal light 21 b.Further, below the respective signal lights, respective arrow signallights 22 r, 22 f, and 22 l for right turn, straight, and left turn areprovided. Due to installation in accordance with the road situation,only one or two of the respective arrow signal lights 22 r, 22 f, and 22l may be installed.

The camera controller 13 reads the average luminance value of the imagedata processed in the image recognition processor 12, sets the exposuretime (shutter speed) for the optimum brightness in capturing an image ofthe next frame, and performs exposure control of the respective cameras2 a and 2 b based on the exposure time. The exposure time is read alsoby the arrow signal recognizer 14.

The arrow signal recognizer 14 checks whether or not lighting of the redsignal light 21 r of the traffic light 21 is detected based on the imagedata processed in the image recognition processor 12 and, in the casewhere lighting is detected, checks whether or not an arrow signal isprovided to the traffic light.

Arrow signal recognition processing executed by the arrow signalrecognizer 14 is specifically processed according to an arrow signalrecognition processing routine shown in FIGS. 2 and 3.

The routine is initiated in synchronization with image information forevery frame transmitted from the image recognition processor 12. First,in step S1, whether or not lighting of the red signal light 21 r of thetraffic light 21 is detected is checked. In the case where lighting isdetected, the routine proceeds to step S2. In the case where lighting isnot detected, the routine is exited to prepare for detection of lightingof the red signal light 21 r in the next frame.

The size (diameter) of the respective signal lights 21 r, 21 y, and 21 bis given in advance. Thus, if the distance (signal light distance) Lsfrom the vehicle 1 to the red signal light 21 r and the actual diameterof the red signal light 21 r are figured out as shown in FIG. 5, thesignal light can be recognized by pattern matching of an image formed inthe main camera 2 a and a template of the signal light set in advance.Further, based on the color feature amount (predetermined RGB ratio) orluminance of the red signal light 21 r, the signal light being lit,i.e., the red signal light 21 r that is the lit signal light, can berecognized.

When it is determined that the red signal light 21 r being lit isdetected and the routine proceeds to step S2, arrow signal areas N1 toN6 are set based on the signal light distance Ls, and the routineproceeds to step S3. The processing in step S2 corresponds to processingby the arrow signal area setting module of the present invention.

For the traffic light 21, there is a vertical type shown in FIG. 7B,other than a horizontal type shown in FIG. 7A. In the horizontal trafficlight 21, the arrow signal lights 22 r, 22 f, and 22 l are providedbelow the red signal light 21 r. In the vertical traffic light 21, thearrow signal lights 22 r, 22 f, and 22 l are provided on the right sideof the red signal light 21 r when seen from the front. Thus, incorrespondence with the arrangements of the arrow signal lights 22 r, 22f, and 22 l, the arrow signal areas N1 to N6 are set below and on theright side, when seen from the front, of the red signal light 21 r. Inthis case, the size (diameter) of the arrow signal light isapproximately the same as the red signal light 21 r, and the approximateinterval of the respective arrow signal lights 22 r, 22 f, and 22 l aregiven. Further, the respective arrow signal lights 22 r, 22 f, and 22 lare provided in positions apart by a distance given in advance from thecorresponding signal lights 21 r, 21 y, and 21 b.

Thus, based on the distance from the vehicle 1 to the red signal light21 r, the actual diameter of the red signal light 21 r, the interval ofthe respective arrow signal lights 22 r, 22 f, and 22 l, and theprovided position of the arrow signal lights 22 r, 22 f, and 22 l withreference to the respective signal lights 21 r, 21 y, and 21 b, theactual distance per pixel pitch on an imaging surface of the main camera2 a is calculated, and the respective arrow signal areas N1 to N6 areset with reference to the red signal light 21 r (see FIG. 6).

Then, the routine proceeds to step S3 and, steps S3 to S17 are repeatedto check whether or not an arrow signal light is present in any one ofthe respective arrow signal areas N1 to N6. First, in step S3, an arrowsignal area Ni is set. The initial value of the arrow signal area Ni is0 and is incremented every time the routine is repeated (Ni←N(i+1) wherei equals 1 to 6). Thus, in this example, search is first started fromthe arrow signal area N1 at the left end, and the next arrow signal areaN2 and the arrow signal area N3 are searched in order, as shown in FIG.6. The arrow signal areas N1 to N3 correspond to the respective arrowsignal lights 22 l, 22 f, and 22 r arranged in the horizontal trafficlight 21 shown in FIG. 7A.

Next, the arrow signal area N4 on the immediate right of the detectedred signal light 21 r when seen from the front is searched, and thearrow signal area N5 and the arrow signal area N6 below are searched inorder. The arrow signal areas N4 to N6 correspond to the respectivearrow signal lights 22 r, 22 f, and 22 l arranged in the verticaltraffic light 21 shown in FIG. 7B.

Then, when the routine proceeds to step S4, pixels in the set arrowsignal area Ni (e.g., area N1) are searched to extract a color toneeffective pixel. That is, in the case where the arrow signal light isLED, the luminance is approximately constant in a lit state, and theedge is detected clearly. Therefore, the color feature amount of thearrow signal is large, and thus most of the color tone effective pixelscan be set directly as arrow effective pixels. The color feature amountis an image feature amount relating to color information and isrepresented by pixel values of respective R, G, and B components of apixel in question.

In this example, the reference for determining the magnitude of thecolor feature amount is set as follows.

G>150

B>0.6G

R<0.5G

A pixel in which the pixel value is exceeded is determined as having alarge color feature amount.

The color tone effective pixel refers to a pixel of which the pixelvalue is smaller than the color feature amount described above but thatcan be assumed as being lit from the color tone (tint or shade). In thisexample, that satisfying two conditions described below is extracted asthe color tone effective pixel. The arrow effective pixel numberdescribed later refers to the remaining number of pixels when the numberof pixels (color tone ineffective pixels) that have been temporarilyassumed a as color tone effective pixel but become ineffective inrelation to luminance information or the like of the vicinity issubtracted from the color tone effective pixel number. Thus, in the caseof an LED arrow signal light, the color tone effective pixel numberapproximately equals the arrow effective pixel number, since there arevery few color tone ineffective pixels.

The determination conditions for assuming a color tone effective pixelare as follows.

(1) The distance from the vehicle 1 to the arrow signal light is thesame as the signal light distance Ls.(2) The RGB ratio satisfies a predetermined condition with reference toluminance information of the vicinity.

The condition of (1) is to confirm provision to the same traffic light21. The condition of (2) is to determine whether or not an arrow signallight, e.g., a light bulb arrow signal light with low luminance and ablurry outline, is lit. Specifically, when the red signal light 21 r islit, each pixel within each arrow signal area Ni is searched with thecomparison reference being the luminance information (RGB ratio) of thesignal light (e.g., yellow signal light 21 y) that is unlit in the sametraffic light 21. That is, since the lit arrow signal light isconsidered to be brighter than at least the unlit signal light (e.g.,yellow signal light 21 y), a pixel of which the detected luminance isbrighter than the luminance information of the unlit signal light isextracted as the color tone effective pixel.

A pixel satisfying the conditions (1) and (2) described above isextracted as the color tone effective pixel. For example, when theluminance information (RGB ratio) of respective pixels in the arrowsignal area Ni set in the imaging surface and a threshold set in advanceare compared for binarization into color tone effective pixels (OK) andineffective pixels (NG) as shown in FIG. 8A, color tone effective pixelsas shown in FIG. 8B are extracted.

Then, when the routine proceeds to step S5, a number (color toneeffective pixel number) IOK of the color tone effective pixels extractedwithin the arrow signal area Ni is counted, and the routine proceeds tostep S6. The processing in steps S4 and S5 correspond to processing bythe color tone effective pixel searching module of the presentinvention.

In step S6, the color tone ineffective pixel is searched for in therespective color tone effective pixels extracted pixel by pixel in thearrow signal area Ni, based on pixel information of vicinity. That is,as described above, it is conceivable that a pixel assumed as the colortone effective pixel is a pixel that is actually unlit but extractederroneously, due to low luminance or a blurry outline.

In the case where one or both of conditions of (1) and (2) describedbelow apply, the color tone effective pixel is set as a color toneineffective pixel.

(1) In the case where a difference AG in G values of a pixel extractedas the color tone effective pixel and a pixel in the vicinity that isextracted as the color tone effective pixel is within a range set inadvance, i.e., a large difference in the G values is absent, the pixelin the vicinity is set as a color tone ineffective pixel.(2) In the case where a pixel extracted as the color tone effectivepixel is neighbored by ineffective pixels (NG), the color tone effectivepixel is set as a color tone ineffective pixel.

Specifically, regarding the condition shown in (1), the G value of thecolor tone effective pixel and the G value of the color tone effectivepixel in the vicinity are compared, since the color of the arrow signallight is green and is most significant in the G value. In the case wherea large difference in the G values is absent, the luminance is uniformthroughout, and there is a possibility of an erroneous determination foran arrow signal. Therefore, such a pixel in the vicinity is set as acolor tone ineffective pixel. For example, in the case where focus is ona pixel in the center as shown in FIG. 9A, difference (edge difference)between the G value of this pixel and the G value of pixels on theimmediate right and below is almost absent, and it is presumed that theinfluence of the luminance of the color tone effective pixel in thecenter is merely causing a blur. Therefore, the pixels on the immediateright and below are set as a color tone ineffective pixel. For the Gvalue of pixels on the immediate left and above, there is a differencewith respect to the G value of the pixel in the center, and it ispresumed that the influence of blur is absent. Therefore, the color toneeffective pixel is left as is. Thus, in FIG. 9A, the pixels on theimmediate right and below are color tone ineffective pixels, and thepixels on the immediate left and above are color tone effective pixels.

Regarding the condition of (2), the color tone effective pixels areusually detected consecutively to some extent. In the case where thesurrounding pixels in the vicinity are ineffective pixels (NG),consecutiveness is lacking, and there is a high possibility that a colortone effective pixel is extracted erroneously. Therefore, such a pixelis set as a color tone ineffective pixel.

Then, when the routine proceeds to step S7, a number (color toneineffective pixel number) ING of the color tone ineffective pixels setin step S6 described above is counted, and the routine proceeds to stepS8. The processing in step S7 corresponds to processing by the colortone ineffective pixel searching module of the present invention.

In step S8, the arrow effective pixel number YOK is calculated from thedifference between the color tone effective pixel number IOK and thecolor tone ineffective pixel number ING (YOK←IOK-ING), and the routineproceeds to step S9. The processing in step S8 corresponds to processingby the arrow effective pixel number calculating module of the presentinvention.

In step S9, an arrow effective pixel threshold SLY is obtained based onthe signal light distance Ls. That is, in the case where the signallight distance Ls is large, the number of pixels for an image of thearrow signal light that is formed on the imaging surface is small, thesignal light distance Ls decreases as the vehicle 1 approaches the redsignal light 21 r, and the arrow signal area Ni (where i equals 1 to 6)is set accordingly. Therefore, the arrow effective pixel threshold SLYfor determining whether or not an arrow signal is lit is set to a largervalue as the signal light distance Ls decreases.

In step S10, the arrow effective pixel number YOK and the arroweffective pixel threshold SLY are compared. In the case where it isdetermined that YOK>SLY, it is determined that an arrow pixel is presentin the corresponding arrow signal area Ni, and the routine proceeds tostep S11. In the case where YOK SLY, it is determined that an arrowpixel is absent, and it branches off to step S12. The processing in stepS10 corresponds to processing by the arrow pixel determining module ofthe present invention.

In step S11, the arrow effective point PNiOK relating to thecorresponding arrow signal area Ni is updated by incrementing thecurrently held arrow effective point PNiOK(n-1) (PNiOK←PNIOK(n-1)+1),and the routine proceeds to step S13. When the routine proceeds to stepS12 from step S10, the arrow ineffective point PNiNG relating to thecorresponding arrow signal area Ni is updated by incrementing thecurrently held arrow effective point PNiNG(n-1) (PNiNG←PNiNG(n-1)+1),and the routine proceeds to step S15. Thus, addition is performed forevery flowchart, i.e., for every frame, in one of the respective pointsPNiOK and PNiNG described above. Note that (n-1) is a reference signshowing the value before update. The processing in step S11 and step S12described later correspond to processing by the arrow point addingmodule of the present invention.

When the routine proceeds to step S13 from step S11, the arrow effectivepoint PNiOK and an arrow effective threshold SLOK set in advance arecompared. When the routine proceeds to step S15 from step S12, the arrowineffective point PNiNG and an arrow ineffective threshold SLNG set inadvance are compared. As described above, the respective points PNiOKand PNiNG are values of which one is added for every frame from when thetraffic light 21 ahead is recognized by the onboard camera 2 mounted tothe vehicle 1 and the arrow signal processing routine is started.

Thus, since the precision in recognizing lighting of an arrow signallight increases as the vehicle 1 approaches the traffic light 21,setting the respective thresholds SLOK and SLNG to a large valueincreases the precision of detection of the arrow signal light, but thetime required for detection increases. Setting the thresholds to SLOKand SLNG a small value reduces the time required for detection, but theprecision of detection decreases. Therefore in this example, values withwhich a certain extent of precision of detection can be obtained and thetime required for detection does not increase are obtained from anexperiment or the like and set as the thresholds SLOK and SLNG. In thisexample, setting is such that SLOK and SLNG approximately equal 15 to 20in the case where the frame rate is 30 (fps).

In the case where it is determined in step S13 that PNiOK>SLOK, i.e., alit arrow signal light is present in the corresponding arrow signal areaNi, the routine proceeds to step S14 to set the arrow signal lightdetermination flag FNi (where i equals 1 to 6), and the routine proceedsto step S17. In the case where PNIOK≦SLOK, the threshold SLOK is notreached, and therefore it jumps to step S17.

In the case where it is determined in step S15 that PNiNG>SLNG, i.e., alit arrow signal light is absent or an arrow signal light itself isabsent in the corresponding arrow signal area Ni, the routine proceedsto step S16 so that both of the points PNiOK and PNiNG are cleared(PNiOK←0 and PNiNG←0), and the routine proceeds to step S17. Thus, inthe case where the same traffic light 21 is recognized at the time ofexecuting the next routine, detection of an arrow signal light isperformed again for the corresponding traffic light 21. The processingin steps S13, S14, S15, and S16 correspond to processing by the arrowsignal determining module of the present invention.

When the routine proceeds to step S17 from any one of steps S13 to S16,whether or not the value Ni of the arrow signal area Ni has reached N6,i.e., whether or not search of all of the arrow signal areas N1 to N6(see FIG. 6) has ended, is checked. In the case where it has not yetended (Ni≠N6), it returns to step S3, and search of the next arrowsignal area N(i+1) is performed. In the case where search of all of thearrow signal areas Ni has ended (Ni=N6), the value of the arrow signalarea Ni is cleared (Ni←N0) in step S18, and the routine proceeds to stepS19.

In step S19, the value of the arrow signal light determination flag FNiof all of the arrow signal areas Ni (where i equals 1 to 6) is checked.In the case where at least one arrow signal light determination flag FNiis set, the routine proceeds to step S20 so that an arrow recognitionflag FY is set (FY←1), and the routine is exited. In the case where anarrow signal light determination flag FNi is not set, it branches off tostep S21 so that the arrow recognition flag FY is cleared (FY←), and theroutine is exited.

By reading the value of the arrow recognition flag FY in a traffic lightrecognition device that recognizes the lit color (red yellow, or green)of a traffic light, for example, information on whether or not an arrowsignal light is provided to the recognized traffic light can beobtained.

In this manner, in this example, the arrow signal areas N1 to N6 are setin the vicinity of and with reference to the red signal light 21 r ofthe traffic light 21 of which an image is formed on the imaging surfaceprovided to the imaging element of the onboard camera 2, the color toneeffective pixel in the respective arrow signal areas N1 to N6 isextracted, the color tone ineffective pixel is then extracted based onpixel information of the vicinity of the extracted color tone effectivepixel, and the arrow effective pixel number YOK is detected from thedifference between the number of the extracted color tone effectivepixels IOK and the number of color tone ineffective pixels ING.Therefore, whether or not an arrow signal is present in each arrowsignal area Ni can be detected with high precision.

Further, in the case where the number YOK of the arrow effective pixelsextracted for each arrow signal area Ni exceeds the arrow effectivepixel threshold SLY set for each frame based on the signal lightdistance Ls, the arrow effective point PNiOK(n-1) is incremented andupdated. In the case where the updated arrow effective point PNiOKexceeds the arrow effective threshold SLOK, it is determined that anarrow signal light is present. Therefore, an arrow signal light can berecognized accurately.

In the case where the arrow effective pixel number YOK is less than orequal to the arrow effective pixel threshold SLY, the arrow ineffectivepoint PNiNG(n-1) is incremented and updated. In the case where theupdated arrow ineffective point PNiNG exceeds the arrow ineffectivethreshold SLNG, the two points PNiOk and PNiNG are cleared, anddetection of an arrow signal light is performed again. Therefore, thefrequency of an erroneous detection is reduced, and a higher precisionof detection of an arrow signal light can be obtained by thereperformance.

The present invention is not limited to the example described above. Forexample, setting the arrow signal light determination flag FNi in stepS14 is performed for each arrow signal area Ni (where i equals 1 to 6).Therefore, it is also possible to specify to which signal light an arrowsignal light is adjacent, on the basis of the value of the arrow signallight determination flag FNi.

1. An arrow signal recognition device comprising: an onboard camera thatcaptures an image of a driving environment ahead of a vehicle equippedwith the arrow signal recognition device; an image recognition processorthat recognizes a lit signal light provided to a traffic light andcalculates a signal light distance between the lit signal light and thevehicle, based on one frame of the image captured by the onboard camera;and an arrow signal detector that detects an arrow signal light providedto the traffic light, wherein the arrow signal detector includes: anarrow signal area setting module that sets an arrow signal area withreference to the lit signal light on the basis of the signal lightdistance; a color tone effective pixel searching module that searchesfor a color tone effective pixel assumed as being lit from color toneswithin an arrow signal area set by the arrow signal area setting moduleand counts the number of the color tone effective pixels; a color toneineffective pixel searching module that compares each color toneeffective pixel retrieved by the color tone effective pixel searchingmodule and a pixel in a vicinity of the each color tone effective pixel,searches the color tone effective pixel for a color tone ineffectivepixel, and counts the number of the color tone ineffective pixels; anarrow effective pixel number calculating module that calculates an arroweffective pixel number by subtracting the number of color toneineffective pixels retrieved by the color tone ineffective pixelsearching module from the number of color tone effective pixelsretrieved by the color tone effective pixel searching module; and anarrow pixel determining module that determines an arrow pixel to bepresent in a case where the arrow effective pixel number calculated bythe arrow effective pixel number calculating module exceeds an arroweffective pixel threshold set on the basis of the signal light distance.2. The arrow signal recognition device according to claim 1, wherein thecolor tone effective pixel searching module assumes the color toneeffective pixel in a case where a distance to the arrow signal lightfrom the vehicle is identical to the signal light distance and apredetermined condition of an RGB ratio with reference to luminanceinformation on an unlit signal light provided to the same traffic lightis satisfied.
 3. The arrow signal recognition device according to claim1, wherein, in a case where a difference in G value between the colortone effective pixel and a color tone effective pixel in the vicinity ofthe color tone effective pixel falls under a narrow range set inadvance, the color tone ineffective pixel searching module sets thecolor tone effective pixel in the vicinity to a color tone ineffectivepixel.
 4. The arrow signal recognition device according to claim 2,wherein, in a case where a difference in G value between the color toneeffective pixel and a color tone effective pixel in the vicinity of thecolor tone effective pixel falls under a narrow range set in advance,the color tone ineffective pixel searching module sets the color toneeffective pixel in the vicinity to a color tone ineffective pixel. 5.The arrow signal recognition device according to claim 1, wherein, in acase where the color tone effective pixel is neighbored by anineffective pixel, the color tone ineffective pixel searching modulesets the color tone effective pixel to a color tone ineffective pixel.6. The arrow signal recognition device according to claim 2, wherein, ina case where the color tone effective pixel is neighbored by anineffective pixel, the color tone ineffective pixel searching modulesets the color tone effective pixel to a color tone ineffective pixel.7. The arrow signal recognition device according to claim 3, wherein, ina case where the color tone effective pixel is neighbored by anineffective pixel, the color tone ineffective pixel searching modulesets the color tone effective pixel to a color tone ineffective pixel.8. The arrow signal recognition device according to claim 1, wherein thearrow signal detector further includes: an arrow point adding modulethat increments an arrow effective point updated at a time ofcomputation for every frame in a case where the arrow pixel determiningmodule has determined an arrow pixel to be present; and an arrow signaldetermining module that determines a lit arrow signal to be present in acase where the arrow effective point set by the arrow point addingmodule exceeds an arrow effective threshold set in advance.
 9. The arrowsignal recognition device according to claim 2, wherein the arrow signaldetector further includes: an arrow point adding module that incrementsan arrow effective point updated at a time of computation for everyframe in a case where the arrow pixel determining module has determinedan arrow pixel to be present; and an arrow signal determining modulethat determines a lit arrow signal to be present in a case where thearrow effective point set by the arrow point adding module exceeds anarrow effective threshold set in advance.
 10. The arrow signalrecognition device according to claim 3, wherein the arrow signaldetector further includes: an arrow point adding module that incrementsan arrow effective point updated at a time of computation for everyframe in a case where the arrow pixel determining module has determinedan arrow pixel to be present; and an arrow signal determining modulethat determines a lit arrow signal to be present in a case where thearrow effective point set by the arrow point adding module exceeds anarrow effective threshold set in advance.
 11. The arrow signalrecognition device according to claim 4, wherein the arrow signaldetector further includes: an arrow point adding module that incrementsan arrow effective point updated at a time of computation for everyframe in a case where the arrow pixel determining module has determinedan arrow pixel to be present; and an arrow signal determining modulethat determines a lit arrow signal to be present in a case where thearrow effective point set by the arrow point adding module exceeds anarrow effective threshold set in advance.